Capturing light-field images with uneven and/or incomplete angular sampling

ABSTRACT

A light-field camera may generate four-dimensional light-field data indicative of incoming light. The light-field camera may have an aperture configured to receive the incoming light, an image sensor, and a microlens array configured to redirect the incoming light at the image sensor. The image sensor may receive the incoming light and, based on the incoming light, generate the four-dimensional light-field data, which may have first and second spatial dimensions and first and second angular dimensions. The first angular dimension may have a first resolution higher than a second resolution of the second angular dimension.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 62/166,595 for “Capturing Light-field Images withUneven and/or Incomplete Angular Sampling” (Atty. Docket No.LYT222-PROV), filed May 26, 2015, the disclosure of which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present document relates to the use of multiple light-field camerasto capture images with uneven and/or irregular angular sampling.

BACKGROUND

Light-field cameras may be used to capture a four-dimensional (4D)light-field image, with two spatial dimensions, x and y, and two angulardimensions, u and v. In many light-field cameras, a plenoptic microlensarray and a single photosensor, containing a two-dimensional (2D) arrayof pixels, are used. These plenoptic light-field cameras capture imagedata that may be much more versatile than traditional two-dimensionalimage data. In particular, a light-field image may be processed tocreate a set of virtual views, in which focus distance, center ofperspective, depth-of-field, and/or other virtual-camera parameters arevaried within ranges enabled by the data in the four-dimensionallight-field image. Further, the light-field data may be analyzed tocalculate a depth map and/or analysis information.

One drawback of existing plenoptic cameras is that virtual views haverelatively low resolution. In order to capture the four-dimensionallight-field, the spatial resolution is decreased significantly in orderto increase the resolution of the angular dimensions. In manyapplications, the output resolution of virtual views may be too low forwidespread adoption.

However, depending on the use case, a complete and even sampling in thetwo angular dimensions may not be required. In many cases, the desiredoutput from the light-field camera may be a combination ofhigh-resolution image data and depth data, often in the form of a depthmap or a set of three-dimensional (3D) points (a point cloud). In someuse cases, the desired output may be high-resolution depth data.

SUMMARY

Various embodiments of the described system capture light-field imageswith uneven and/or incomplete angular sampling. Such embodiments mayincrease spatial and/or output resolution, increase the quality of thedepth data, extend the refocusable range of the system, and/or increasethe optical baseline. Some of the embodiments utilize novel exit pupilshapes and/or configurations of the microlens array.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate several embodiments. Together withthe description, they serve to explain the principles of theembodiments. One skilled in the art will recognize that the particularembodiments illustrated in the drawings are merely exemplary, and arenot intended to limit scope.

FIG. 1 is a diagram of a plenoptic light-field camera, according to oneembodiment.

FIGS. 2A-2E are diagrams of various aspects of the plenoptic light-fieldcamera of FIG. 1.

FIG. 3 shows a cross-sectional view of a double Gauss lens design,including an aperture stop, according to one embodiment.

FIG. 4 shows a portion of an image captured with a microlens arraypaired with a circular aperture.

FIG. 5 shows a portion of an image captured with a microlens arraypaired with a rectangular or even square aperture.

FIG. 6 shows a portion of an image captured with a microlens arraypaired with a generally circular aperture.

FIGS. 7A-7C show the relationships between the exit pupil, microlensarray, and disk images, according to certain embodiments.

FIGS. 8A and 8B show exemplary embodiments with exit pupils of highaspect ratios, with nonzero rotational offsets applied between the exitpupils and the microlens arrays and square packing of microlenses,according to certain embodiments.

FIGS. 9A and 9B show exemplary embodiments with exit pupils of highaspect ratios, with nonzero rotational offsets applied between the exitpupils and the microlens arrays and hexagonal packing of microlenses,according to certain embodiments.

FIGS. 10A, 10B and 10C show captured disk images using a hexagonallypacked microlens array and three different exit pupil configurations,according to certain embodiments.

FIGS. 11A and 11B show exemplary embodiments with the exit pupils ofhigh aspect ratios, with nonzero rotational offsets applied between theexit pupils and the microlens arrays and hexagonal packing ofmicrolenses, according to another embodiment.

FIG. 12 shows an exemplary embodiment with an exit pupils of high aspectratio, with nonzero rotational offsets applied between the exit pupilsand the microlens arrays and hexagonal packing of microlenses, accordingto another embodiment.

FIGS. 13A and 13B show a virtual view in the form of an image, and depthdata in the form of a depth map corresponding to the image,respectively, according to one embodiment.

FIGS. 14A and 14B conceptually show a discontinuous aperture with threesections, according to one embodiment.

FIG. 15 shows the relationship between the exit pupil, microlens array,and disk images when the exit pupil has a high aspect ratio and themicrolens array uses lenses of the same aspect ratio using rectangularpacking.

FIG. 16 shows the relationship between the exit pupil, microlens array,and disk images when the exit pupil has a high aspect ratio and themicrolens array uses cylindrical lenses, according to one embodiment.

FIGS. 17A-17D show exemplary microlens packing arrangements.

FIG. 18A conceptually shows a discontinuous exit pupil of two parts,according to one embodiment. FIG. 18B shows how segments of a disk imagecreated from a square packed microlens array and an exit pupil of thistype may tessellate.

FIGS. 19A and 19B show two exemplary embodiments on a square packedmicrolens array, using an exit pupil that is discontinuous in twodimensions, according to one embodiment.

FIGS. 20A-20D conceptually show an embodiment that may use a variety ofmasks to create a light-field camera with different configurableproperties, but without changing the microlens array, photosensor,and/or any additional aspect of the objective lens, according to oneembodiment.

FIG. 21 depicts a portion of a light-field image.

FIG. 22 depicts an example of an architecture for implementing themethods of the present disclosure in a light-field capture device,according to one embodiment.

FIG. 23 depicts an example of an architecture for implementing themethods of the present disclosure in a post-processing systemcommunicatively coupled to a light-field capture device, according toone embodiment.

FIG. 24 depicts an example of an architecture for a light-field camerafor implementing the methods of the present disclosure according to oneembodiment.

DETAILED DESCRIPTION

Multiple methods for capturing image and/or video data in a light-fieldvolume and creating virtual views from such data are described. Thedescribed embodiments may provide for capturing continuous or nearlycontinuous light-field data from many or all directions facing away fromthe capture system, which may enable the generation of virtual viewsthat are more accurate and/or allow viewers greater viewing freedom.

Definitions

For purposes of the description provided herein, the followingdefinitions are used:

-   -   Alias-free resolution: a resolution equal to the number of        microlenses in the plenoptic microlens array; two-dimensional        images created at this resolution from a light field image        typically do not contain objectionable processing artifacts.    -   Aperture stop: The element, be it the rim of a lens or a        separate diaphragm, which determines the amount of light        reaching the image. In this disclosure, “aperture stop” and        “aperture” may be used interchangeably.    -   Conventional image: an image in which the pixel values are not,        collectively or individually, indicative of the angle of        incidence at which light is received by a camera.    -   Depth: a representation of distance between an object and/or        corresponding image sample and a microlens array of a camera.    -   Depth data: any depth or three-dimensional information that may        be generated from light field data, which may include, but is        not limited to, a depth map, a three-dimensional point cloud,        and/or a three-dimensional mesh.    -   Depth map: a two-dimensional array of depth values, which may be        calculated from a light field image. See also “depth data.”    -   Depth-of-field: the range of object distances for which a        projected image (especially a virtual view) is sharp to some        sufficient degree.    -   Disk: a region in a light-field image that is illuminated by        light passing through a single microlens; may be circular or any        other suitable shape.    -   Elliptical packing: a packing pattern that tessellates stretched        hexagonal regions onto a stretched hexagonal lattice. In this        disclosure, elliptical packing is used to describe the pattern        of the microlens elements in the microlens array. An example of        this packing is shown in FIG. 17D.    -   Entrance pupil: in an optical system, the optical image of the        physical aperture stop, as seen through the front of the lens        system. The geometric size, location, and angular acceptance of        the entrance pupil acts as the camera's window of view into the        world.    -   Exit pupil: the exit pupil is the image of the aperture stop as        seen from an axial point on the image plane through the        interposed lenses, if there are any. In a light-field camera,        the image plane is best thought of as the active surface of the        microlens array.    -   F-Number (f/#): focal length divided by entrance pupil size. In        this document, the entrance pupil size used in the calculation        of f/# is considered to be the width or height for a square, the        inner diameter for a hex, or the diameter of a circle. The f/#        of the microlens array may be considered to be the lens pitch        (equal to the distance between the centers of neighboring lens        elements) divided by the distance between the microlens array        and the sensor surface.    -   Hexagonal packing: a packing pattern that tessellates hexagonal        regions onto a hexagonal lattice. In this disclosure, hexagonal        packing is a specific type of elliptical packing where the        hexagon is regular. In this disclosure, hexagonal packing is        used to describe the pattern of the microlens elements in the        microlens array. An example of this packing is shown in FIG.        17C.    -   Image: a two-dimensional array of pixel values, or pixels, each        specifying a color.    -   Input device: any device that receives input from a user.    -   Light-field camera: any camera or device capable of capturing        light-field images.    -   Light-field coordinate, or “four-dimensional light-field        coordinate”: for a single light-field camera, the        four-dimensional coordinate (for example, x, y, u, v) used to        index a light-field sample captured by a light-field camera, in        which (x, y) may be the spatial coordinate representing the        intersection point of a light ray with a microlens array, and        (u, v) may be the angular coordinate representing an        intersection point of the light ray with an aperture plane.    -   Light-field data: data indicative of the angle of incidence at        which light is received by a camera.    -   Light-field image: an image that contains a representation of        light-field data captured at the sensor, which may be a        four-dimensional sample representing information carried by ray        bundles received by a single light-field camera.    -   Main lens, or objective lens: a lens or set of lenses that        directs light from a scene toward an image sensor.    -   Microlens: a small lens, typically one in an array of similar        microlenses.    -   Microlens array: an array of microlenses arranged in a        predetermined pattern.    -   MLA-to-exit pupil rotation: the rotation, in degrees, of the        long axis of the exit pupil relative to a primary axis of the        microlens array.    -   MLA-to-sensor rotation: the rotation, in degrees, of a primary        axis of the microlens array relative to a primary axis of the        photosensor array.    -   Narrow F/#: For an irregular exit pupil, the f/# based on the        short axis of the exit pupil.    -   Optical baseline: the size of the entrance pupil, as measured        across some axis. In this disclosure, the optical baseline        refers to the measurement across the long axis, unless otherwise        specified. Larger optical baselines equate to greater disparity        between the opposing sides of the entrance pupil. The greater        disparity may increase the accuracy of certain types of        calculations, particularly in the generation of depth data.    -   Packing, or packing arrangement: the manner in which microlenses        are arranged to form a microlens array    -   Plenoptic light-field camera: a type of light-field camera that        employs a microlens-based approach in which a plenoptic        microlens array is positioned between the objective lens and the        photosensor.    -   Plenoptic microlens array: a microlens array in a plenoptic        camera that is used to capture directional information for        incoming light rays, with each microlens creating an image of        the aperture stop of the objective lens on the surface of the        image sensor.    -   Processor: any processing device capable of processing digital        data, which may be a microprocessor, ASIC, FPGA, or other type        of processing device.    -   Ray bundle, ray, or bundle: a set of light rays recorded in        aggregate by a single pixel in a photosensor.    -   Rectangular packing: a packing pattern that tessellates        rectangular regions onto a rectangular grid. In this disclosure,        rectangular packing is used to describe the pattern of the        microlens elements in the microlens array. An example of this        packing is shown in FIG. 17B.    -   Segment, or image segment: a single image of the exit pupil,        viewed through a plenoptic microlens, and captured by a region        on the surface of an image sensor.    -   Sensor, photosensor, or image sensor: a light detector in a        camera capable of generating images based on light received by        the sensor.    -   Subview: the view or image from an individual view in a        light-field camera (a subaperture image in a plenoptic        light-field camera, or an image created by a single objective        lens in an objective lens array in an array light-field camera).    -   Super resolution: resolutions higher than the alias-free        resolution. Certain image processing techniques can        significantly increase the resolution of reconstructed        two-dimensional images, but may introduce objectionable visual        artifacts.    -   Virtual view: a two-dimensional image created by processing a        light field image based on various parameters. Virtual view        types include, but are not limited to, refocused images and        extended depth of field (EDOF) images.    -   Wide F/#: For an irregular exit pupil, the f/# based on the long        axis of the exit pupil.

In addition, for ease of nomenclature, the term “camera” is used hereinto refer to an image capture device or other data acquisition device.Such a data acquisition device can be any device or system foracquiring, recording, measuring, estimating, determining and/orcomputing data representative of a scene, including but not limited totwo-dimensional image data, three-dimensional image data, and/orlight-field data. Such a data acquisition device may include optics,sensors, and image processing electronics for acquiring datarepresentative of a scene, using techniques that are well known in theart. One skilled in the art will recognize that many types of dataacquisition devices can be used in connection with the presentdisclosure, and that the disclosure is not limited to cameras. Thus, theuse of the term “camera” herein is intended to be illustrative andexemplary, but should not be considered to limit the scope of thedisclosure. Specifically, any use of such term herein should beconsidered to refer to any suitable device for acquiring image data.

In the following description, several techniques and methods forprocessing light-field images are described. One skilled in the art willrecognize that these various techniques and methods can be performedsingly and/or in any suitable combination with one another. Further,many of the configurations and techniques described herein areapplicable to conventional imaging as well as light-field imaging. Thus,although the following description focuses on light-field imaging, allof the following systems and methods may additionally or alternativelybe used in connection with conventional digital imaging systems. In somecases, the needed modification is as simple as removing the microlensarray from the configuration described for light-field imaging toconvert the example into a configuration for conventional image capture.

Architecture

In at least one embodiment, the system and method described herein canbe implemented in connection with light-field images captured bylight-field capture devices including but not limited to those describedin Ng et al., Light-field photography with a hand-held plenoptic capturedevice, Technical Report CSTR 2005-02, Stanford Computer Science.Referring now to FIG. 22, there is shown a block diagram depicting anarchitecture for implementing the method of the present disclosure in alight-field capture device such as a camera 2200. Referring now also toFIG. 23, there is shown a block diagram depicting an architecture forimplementing the method of the present disclosure in a post-processingsystem 2300 communicatively coupled to a light-field capture device suchas a camera 2200, according to one embodiment. One skilled in the artwill recognize that the particular configurations shown in FIGS. 22 and23 are merely exemplary, and that other architectures are possible forcamera 2200. One skilled in the art will further recognize that severalof the components shown in the configurations of FIGS. 22 and 23 areoptional, and may be omitted or reconfigured.

In at least one embodiment, camera 2200 may be a light-field camera thatincludes light-field image data acquisition device 2209 having optics2201, image sensor 2203 (including a plurality of individual sensors forcapturing pixels), and microlens array 2202. Optics 2201 may include,for example, aperture 2212 for allowing a selectable amount of lightinto camera 2200, and main lens 2213 for focusing light toward microlensarray 2202. In at least one embodiment, microlens array 2202 may bedisposed and/or incorporated in the optical path of camera 2200 (betweenmain lens 2213 and image sensor 2203) so as to facilitate acquisition,capture, sampling of, recording, and/or obtaining light-field image datavia image sensor 2203. Referring now also to FIG. 24, there is shown anexample of an architecture for a light-field camera, or camera 2200, forimplementing the method of the present disclosure according to oneembodiment. The figure is not shown to scale. FIG. 24 shows, inconceptual form, the relationship between aperture 2212, main lens 2213,microlens array 2202, and image sensor 2203, as such components interactto capture light-field data for one or more objects, represented by anobject 2401, which may be part of a scene 2402.

In at least one embodiment, camera 2200 may also include a userinterface 2205 for allowing a user to provide input for controlling theoperation of camera 2200 for capturing, acquiring, storing, and/orprocessing image data. The user interface 2205 may receive user inputfrom the user via an input device 2206, which may include any one ormore user input mechanisms known in the art. For example, the inputdevice 2206 may include one or more buttons, switches, touch screens,gesture interpretation devices, pointing devices, and/or the like.

Similarly, in at least one embodiment, post-processing system 2300 mayinclude a user interface 2305 that allows the user to control operationof the system.

In at least one embodiment, camera 2200 may also include controlcircuitry 2210 for facilitating acquisition, sampling, recording, and/orobtaining light-field image data. The control circuitry 2210 may, inparticular, be used to switch image capture configurations in responseto receipt of the corresponding user input. For example, controlcircuitry 2210 may manage and/or control (automatically or in responseto user input) the acquisition timing, rate of acquisition, sampling,capturing, recording, and/or obtaining of light-field image data.

In at least one embodiment, camera 2200 may include memory 2211 forstoring image data, such as output by image sensor 2203. Such memory2211 can include external and/or internal memory. In at least oneembodiment, memory 2211 can be provided at a separate device and/orlocation from camera 2200.

For example, when camera 2200 is in a light-field image captureconfiguration, camera 2200 may store raw light-field image data, asoutput by image sensor 2203, and/or a representation thereof, such as acompressed image data file. In addition, when camera 2200 is in aconventional image capture configuration, camera 2200 may storeconventional image data, which may also be stored as raw, processed,and/or compressed output by the image sensor 2203.

In at least one embodiment, captured image data is provided topost-processing circuitry 2204. The post-processing circuitry 2204 maybe disposed in or integrated into light-field image data acquisitiondevice 2209, as shown in FIG. 22, or it may be in a separate componentexternal to light-field image data acquisition device 2209, as shown inFIG. 23. Such separate component may be local or remote with respect tolight-field image data acquisition device 2209. Any suitable wired orwireless protocol can be used for transmitting image data 2221 tocircuitry 2204; for example, the camera 2200 can transmit image data2221 and/or other data via the Internet, a cellular data network, aWi-Fi network, a Bluetooth communication protocol, and/or any othersuitable means.

Such a separate component may include any of a wide variety of computingdevices, including but not limited to computers, smartphones, tablets,cameras, and/or any other device that processes digital information.Such a separate component may include additional features such as a userinput 2215 and/or a display screen 2216. If desired, light-field imagedata may be displayed for the user on the display screen 2216.

Overview

Light-field images often include a plurality of projections (which maybe circular or of other shapes) of aperture 2212 of camera 2200, eachprojection taken from a different vantage point on the camera's focalplane. The light-field image may be captured on image sensor 2203. Theinterposition of microlens array 2202 between main lens 2213 and imagesensor 2203 causes images of aperture 2212 to be formed on image sensor2203, each microlens in microlens array 2202 projecting a small image ofmain-lens aperture 2212 onto image sensor 2203. These aperture-shapedprojections are referred to herein as disks, although they need not becircular in shape. The term “disk” is not intended to be limited to acircular region, but can refer to a region of any shape.

Light-field images include four dimensions of information describinglight rays impinging on the focal plane of camera 2200 (or other capturedevice). Two spatial dimensions (herein referred to as x and y) arerepresented by the disks themselves. For example, the spatial resolutionof a light-field image with 120,000 disks, arranged in a Cartesianpattern 400 wide and 300 high, is 400×300. Two angular dimensions(herein referred to as u and v) are represented as the pixels within anindividual disk. For example, the angular resolution of a light-fieldimage with 100 pixels within each disk, arranged as a 10×10 Cartesianpattern, is 10×10. This light-field image has a 4-D (x,y,u,v) resolutionof (400,300,10,10). Referring now to FIG. 21, there is shown an exampleof a 2-disk by 2-disk portion of such a light-field image, includingdepictions of disks 2102 and individual pixels 2101; for illustrativepurposes, each disk 2102 is ten pixels 2101 across.

In at least one embodiment, the 4-D light-field representation may bereduced to a 2-D image through a process of projection andreconstruction. As described in more detail in related U.S. Utilityapplication Ser. No. 13/774,971 for “Compensating for Variation inMicrolens Position During Light-Field Image Processing,” (Atty. DocketNo. LYT021), filed Feb. 22, 2013 and issued on Sep. 9, 2014 as U.S. Pat.No. 8,831,377, the disclosure of which is incorporated herein byreference in its entirety, a virtual surface of projection may beintroduced, and the intersections of representative rays with thevirtual surface can be computed. The color of each representative raymay be taken to be equal to the color of its corresponding pixel.

Problem Description

One drawback of many existing plenoptic cameras is relatively low outputresolution. In many such cameras, in order to capture thefour-dimensional light-field, the spatial resolution is decreased verysignificantly in order to increase the resolution of the angulardimensions.

For example, consider the case of a photosensor with a 4000×3000 pixelarray (12 megapixels). Using a standard optical configuration (atwo-dimensional camera), the spatial output resolution is the full 12megapixels. If the same sensor is used in a light-field camera, theconfiguration may include 10 angular samples in each of u and v. In thisconfiguration, the light-field sensor would sample at 400×300×10×10,using the full 12 megapixels. The alias-free spatial resolution ofvirtual views would be 400×300, a very low resolution. As higherresolution is desired for many use cases, including artistic photographyand industrial imaging, it is desirable to find ways to increase theoutput resolution of the system while preserving some or all of theother benefits of light-field images.

Referring to FIG. 1, a plenoptic light-field camera 100 may capture alight-field using an objective lens 110, plenoptic microlens array 120,and photosensor 130. The objective lens 110 may be positioned to receivelight through an aperture (not shown) having an exit pupil. Eachmicrolens in the plenoptic microlens array 120 may create an image ofthe aperture on the surface of the photosensor 130. By capturing dataregarding the vector at which light rays are received by the photosensor130, the plenoptic light-field camera 100 may facilitate the generationof extended depth-of-field images and other processed images based onthe light-field data captured by the plenoptic light-field camera 100.FIG. 1 is a simplified representation for illustrative purposes;light-field camera 100 may include additional components and elementsnot depicted in FIG. 1.

FIGS. 2A through 2E are diagrams of various aspects of the plenopticlight-field camera 100 of FIG. 1. FIG. 2A shows a cross-sectional viewof the plenoptic light-field camera 100, including an aperture stop 200in the objective lens 110. FIG. 2B shows a cross-sectional illustrationof a plenoptic microlens 210 of the plenoptic microlens array 120,central rays 220 passing through the plenoptic microlens 210, and thedisk image 230 generated on the surface of the photosensor 130. FIG. 2Cis a diagram of the aperture stop 200 from a top-down view. FIG. 2D is atop down view of a 2×2 set of plenoptic microlenses 210 with squarepacking. FIG. 2E is a top down view of the disk images 230 generated onthe surface of the photosensor 130.

FIG. 3 shows a cross-sectional view of a double Gauss lens design 300,including an aperture stop 310. The double Gauss lens design 300 is oneof many lens types that may be suitable for use in light-field imaging.As shown, the double Gauss lens design 300 may have a plurality of lenselements 320.

FIGS. 17A through 17D show exemplary lens packing arrangements,according to certain embodiments. FIG. 17A shows a square packingarrangement 1700. FIG. 17B shows a rectangular packing arrangement 1720.FIG. 17C shows a hexagonal packing arrangement 1740. FIG. 17D shows anelliptical packing arrangement 1760.

FIGS. 4, 5 and 6 show exemplary image data captured by a light-fieldcamera, using different aperture shapes and microlens array packing.Various combinations of aperture shapes and microlens array packinglayouts may be used.

Specifically, FIG. 4 is a portion of an image 400 captured with amicrolens array paired with a circular aperture. The microlenses 410 ofthe microlens array may be generally circular in shape to match theshape of the aperture, and may be packed such that each pair of adjacentrows of the microlenses 410 is offset from each other by 50% of thediameter of a microlens 410. This packing may enable the protrudingportions of each microlens 410 to protrude into the empty areas betweenthe adjoining microlenses 410 of each adjacent row. The result may bethat each microlens 410 has six immediate neighbors arranged around themicrolens 410 in a generally hexagonal formation. This packing layoutmay be referred to as “hexagonal packing,” as in the hexagonal packingarrangement 1740 of FIG. 17C.

FIG. 5 is a portion of an image 500 captured with a microlens arraypaired with a rectangular or even square aperture. The microlenses 510of the microlens array may be generally square in shape to match theshape of the aperture, and may be packed in a generally rectangulargrid, with each microlens 510 aligned with its horizontal and verticalneighbors. This packing layout may be referred to as “square packing,”as in the square packing arrangement 1700 of FIG. 17A.

FIG. 6 is a portion of an image 600 captured with a microlens arraypaired with a generally circular aperture. The microlenses 610 of themicrolens array may be generally circular in shape to match the shape ofthe aperture, and may be packed in a generally rectangular grid, witheach microlens 610 aligned with its horizontal and vertical neighbors.Thus, square packing may be used for the microlenses 610 as in the image500, as in the square packing arrangement 1700 of FIG. 17A. The resultmay be that, in the image 600, there is more interstitial “black” spacethan is present in the image 400. However, the horizontal and verticalalignment of the microlenses 610 may provide some computationaladvantages as the image 600 is processed.

Applying specialized processing techniques, such as super resolution,may increase the output resolution of the virtual views. However, whilesuper resolution techniques may be used to increase the outputresolution, the results may still have inadequate resolution. Inaddition, the super resolution techniques often introduce objectionablevisual artifacts. In particular, many existing super resolutiontechniques are dependent on the quality of the depth map. Errors in thedepth map calculated from the light-field data may result in artifacts.

Another method to increase the output resolution of a plenoptic camerais to reduce the number of pixels under each microlens. In the12-megapixel example above, the device may sample at 800×600×5×5. Inthis case, the spatial resolution may be increased, but the angularresolution is decreased. Reducing the angular sampling may also causethe refocusable range and other benefits of the light-field image to bereduced.

An alternative method to increase the output resolution is to increasethe pixel count of the sensor. For example, a 48-megapixel sensor may beused instead of a 12-megapixel sensor. In that case, the 4D samplingresolution may be 800×600×10×10, if all new resolution is allocated tothe spatial dimensions. Alternatively, new resolution may be evenlyallocated approximately evenly across all four dimensions with asampling resolution of 572×428×14×14. While increasing the pixel countimproves output resolution, such an approach significantly increases therequirements of the photosensor, readout, storage, processing and otheraspects of the total image processing system. Further, for a givenphysical area, there are practical limits (including optical aspectssuch as diffraction) to the maximum number of pixels that may beeffectively used.

Novel Exit Pupil Shapes and Microlens Configurations

Various embodiments of the systems and methods described herein capturelight-field images with uneven and/or incomplete angular sampling. Suchembodiments may increase spatial and/or output resolution, increase thequality of the depth data, extend the refocusable range of the system,and/or increase the optical baseline. Some of the embodiments utilizenovel exit pupil shapes and/or configurations of the microlens array.These various approaches can be implemented singly or in any suitablecombination with one another.

Exit Pupils with High Aspect Ratio

In at least one embodiment, the exit pupil contains a long axis and ashort axis. Embodiments of this type may have relatively higher angularsampling in one dimension than in the other, orthogonal dimension.

Embodiments of this type may be advantageous when depth data and/orhigh-resolution virtual view data are desired as output. Generatingdepth data from light-field data is dependent on the algorithm(s) andprocessing selected, but in general a larger optical baseline and/orhigher density angular sampling provides better results. Depth data maybe used to apply effects, modify the image, generate three-dimensionalmodels of objects in the scene, and/or the like.

FIGS. 13A and 13B respectively show a virtual view in the form of animage 1300 and depth data in the form of a depth map 1350. The image1300 may be projected from light-field data. The depth map 1350 may be agrayscale image as shown, and may be generated by processing thelight-field data. In the depth map 1350, dark portions of the imagerepresent surfaces that are closer to the camera, while light portionsindicate surfaces further from the camera.

A standard plenoptic camera system has an alias-free output resolutionapproximately equal to photo_sensor_pixel_count/N². In the limit, anembodiment with an exit pupil with a high aspect ratio may have angularsampling of Nu*Nv, where Nv=1. In that case, the alias-free outputresolution may become photo_sensor_pixel_count/Nu, an inversely linearrelationship with Nu, which may be far preferable to the standardplenoptic relationship that is inversely quadratic with N. As a result,given the same image sensor, configurations may be made that have higherangular sampling (along one axis) with the same alias-free outputresolution, higher alias-free output resolution with the same angularsampling, or a combination of higher alias-free output resolution andhigher angular sampling.

In at least one embodiment, the light-field camera may use cylindricallenses in the microlens array. One example of such an embodiment isshown in FIG. 16.

Referring to FIG. 16, an exemplary exit pupil 1600 is shown, along witha portion of a microlens array 1610 and a portion of a disk image 1620.The exit pupil 1600 may have a high aspect ratio. Specifically, the exitpupil 1600 may be much wider along a long axis than it is tall along ashort axis perpendicular to the long axis. In some embodiments, theaspect ratio of the exit pupil may be N:1.

The microlens array 1610 may be a one-dimensional array of cylindricallenses 1630. The focal length of the cylindrical lenses 1630 may beapproximately equal to the separation between the microlens array andthe photosensor (not shown). The focal length of the microlens array1610 divided by the width of a cylindrical lens 1630 may also besubstantially equivalent to the Wide F/# of the objective lens (notshown). The short axis of the cylindrical lenses 1630 may be parallel tothe long axis of the exit pupil 1600, and this case may be considered tohave an MLA-to-exit pupil rotation of 0°.

The disk image 1620 may have a plurality of segments 1640, each of whichhas a width substantially equal to N pixels. In this configuration, thefour-dimensional sampling rate on a photosensor of W×H pixels may beW/N×H×N×1. This type of camera may be considered a three-dimensionallight-field camera, as one of the angular dimensions contains only asingle sample. Notably, one skilled in the art will recognize that whilethe microlens array 1610 and the exit pupil 1600 may be aligned witheach other to provide a 0° rotational offset between the microlens array1610 and the exit pupil 1600, the sensor (not shown) need not beconstrained to a 0° relative to the microlens array 1610 and/or the exitpupil 1600.

In at least one embodiment in which cylindrical lenses are used in themicrolens array, an anamorphic lens and/or sensor with a high aspectratio may be used. In one particular embodiment, the aperture stop maybe physically circular, square, or otherwise have substantially equalwidth and height. The anamorphic lens may be used to stretch theaperture and image on the sensor, for example by a factor of N. In thiscase, the exit pupil, or the view of the aperture stop as seen from thesensor, may have a long axis and a short axis. The cylindrical microlensarray may be designed so that each disk image is N pixels wide. Further,an ultrawide sensor may be used. For example, if an aspect ratio of W:His desired and the anamorphic lens stretches the image horizontally by afactor of N, then the aspect ratio of the sensor may be WN:H. Notably,the reconstruction processing may reverse the stretching introduced bythe anamorphic lens and output virtual views with an aspect ratio ofW:H.

In some embodiments, the microlens array may use rectangular orelliptical packing of the microlens elements, and may have anMLA-to-exit pupil rotation of 0°. This is shown in FIG. 15, which is aconceptual illustration using a rectangular packing of microlenselements.

Referring to FIG. 15, an exemplary exit pupil 1500 is shown, along witha portion of a microlens array 1510 and a portion of a disk image 1520.The exit pupil 1500 may have a high aspect ratio, like the exit pupil1600 of FIG. 16. Specifically, the exit pupil 1500, the microlenses 1530of the microlens array 1510, and the segments 1540 of the disk image1520 may all have an aspect ratio of c²:1.

In similar embodiments (not shown), the aperture may be physicallycircular, square or otherwise have substantially equal width and height,while the microlens array and disk image segments may have an aspectratio of c²:1. The anamorphic lens may be used to stretch the apertureand image on the sensor, for example by a factor of c². In such cases,the exit pupil, or the view of the aperture stop as seen from thesensor, may have a long axis and a short axis. In at least oneembodiment, a wide sensor may be used such that the virtual views willhave a desired aspect ratio that may be different than the aspect ratioof the two-dimensional image data captured on the photosensor. Oneskilled in the art will recognize that, in any of the embodimentsdescribed herein, the system may also be implemented using verticalconfigurations or any other orientation.

In some embodiments, the exit pupil may have a long axis and a shortaxis, the plenoptic microlens array may use square packing or hexagonalpacking, and the MLA-to-exit pupil rotation may be set to a specificangle so that the disk images tessellate without overlapping. FIGS.7A-7C show the relationships between the exit pupil, microlens array,and disk images in such embodiments.

FIG. 7A shows one typical configuration for a light-field camera. Theexit pupil 700 is circular, the microlenses 712 of the microlens array710 are arranged in a square packing, and the f/# of the main lens isequal or nearly equal to the f/# of the microlens array. In thisconfiguration, the segments 722 of the disk image 720 are tightly packedcircles in a square lattice.

FIG. 7B shows the effect of increasing the size of an exit pupil 730while keeping other aspects unchanged. The f/# of the main lens (notshown) is lower than the f/# of the microlens array 740. In this case,the segments 752 of the disk image 750 show significant overlap, and theimage data may not be usable.

FIG. 7C shows an embodiment of the invention. The exit pupil 760 is aswide as shown in FIG. 7B, but is only half as tall. The microlens array770 is rotated 45°, but otherwise unchanged. In this configuration, thesegments 782 of the disk image 780 do not overlap and have an aspectratio of 2:1.

In a similar embodiment, FIGS. 10A, 10B and 10C show captured images1000, 1030, and 1060, respectively, using a hexagonally packed microlensarray and three different exit pupil configurations. In the image 1000of FIG. 10A, the f/# of the main lens is approximately equal to the f/#of the microlens array and the exit pupil shape is circular. The diskimages 1010 are circular and almost touching. In the image 1030 of FIG.10B, the f/# of the main lens is smaller than the f/# of the microlensarray, and the exit pupil is circular. The disk images 1040 are circularand overlap. The overlapping regions are the brightest portions of theimage. In FIG. 10C, the Wide F/# is smaller than the f/# of themicrolens array, the Narrow F/# is larger than the f/# of the microlensarray, and the long axis of the exit pupil is rotated relative to thehexagonal axes. The disk images 1070 appear rotated and elongated.

Notably, in these embodiments, the sampling along one angular dimensionis greater than in the other angular dimension, but the reduction inspatial resolution may be equal across both spatial dimensions, relativeto the sensor. As an example, the exemplary 4000×3000 pixel sensor, in astandard light-field configuration with N=10, will have a spatialresolution of 4000/N=400 in one dimension and 3000/N=300 in the otherspatial dimension. Using rectangular or elliptical microlens packing,with matched exit pupil shape, may result in a similar tradeoff. If Nx,the number of pixels in a disk image along the x-axis, is 20, and Ny,the number of pixels in a disk image along the y-axis, is 5, then thespatial resolution may be 4000/Nx=200 by 3000/Ny=600. However, in theexample shown in FIG. 7C, the structure of the microlens array 770remains square, and the spatial resolution may remain 400×300 for theexemplary sensor even though Nx is substantially larger than Ny.

In at least one embodiment, the microlens array uses square packing. Atspecific MLA-to-exit pupil rotations, specific aspect ratios may be usedfor the exit pupil and disk images. With square microlens array packing,the following equations describe the system for integral values of k.

angle=90−a tan(k)

width=SQRT(k̂2+1)

height=1/W

aspect_ratio=width:height

where

-   -   “k” is a the sequence number    -   “angle” is the MLA-Exit Pupil rotation, in degrees    -   “width” is the width of the disk image, relative to the width at        k=0    -   “height” is the height of the disk image, relative to the height        at k=0    -   “aspect_ratio” is the aspect ratio of disk images that results        in fully tessellated packing, on the sensor, with no dead space        or overlap

k angle width height aspect ratio 0 90 1 1 1:1 1 45 1.41 0.707 2:1 226.6 2.24 0.447 5:1 3 18.4 3.16 0.316 10:1  4 14.0 4.12 0.243 17:1 

Notably, “k” may be any integer value. Further, “angle” may be anyrotation that results in a similar tessellation pattern in anyorientation. For example, angle, 90°−angle, 90°+angle, 180°−angle,180°+angle, 270°−angle, and 270°+angle may all result in similartessellation patterns.

FIG. 8A shows an embodiment using parameters in the table above wherek=2, with an exit pupil 800 with an aspect ratio of 5:1. The microlensarray 810 is rotated 26.6°. The segments 840 of the disk image 820 havea 5:1 aspect ratio and do not overlap. FIG. 8B shows the configurationwhere k=3, and an exit pupil 850 has an aspect ratio of 10:1. Themicrolens array 860 is rotated 18.4° relative to the exit pupil 850. Thesegments 890 of the disk images 870 have a 10:1 aspect ratio and do notoverlap.

In at least one embodiment, the microlens array uses hexagonal packing.At specific MLA-to-exit pupil rotations, specific aspect ratios may beused for the exit pupil and disk images. With hexagonal microlens arraypacking, the following equations describe the system for integral valuesof k.

angle=a tan(0.5*sqrt(3)/(k+0.5))

width=SQRT((k+0.5)̂2+(0.5*SQRT(3))̂2)

height=SQRT(3)/(W*2)

aspect_ratio=width:height

where

-   -   k is a the sequence number    -   angle is the MLA-Exit Pupil rotation, in degrees    -   width is the width of the disk image, relative to the width at        k=0    -   height is the height of the disk image, relative to the height        at k=0    -   aspect_ratio is the aspect ratio of disk images that results in        fully tessellated packing, on the sensor, with no dead space or        overlap

MLA-Exit k Pupil rotation width height aspect ratio 0 60 1 1 1:1 1 301.73 0.5 3.5:1   2 19.1 2.65 0.33 8:1 3 13.9 3.61 0.24 15:1  4 10.9 4.580.19 24:1 

Notably, “k” may be any integer value. Further, “angle” may be anyrotation that results in a similar tessellation pattern in anyorientation. For example, angle, 90°−angle, 90°+angle, 180°−angle,180°+angle, 270°−angle, and 270°+angle may all result in similartessellation patterns.

FIGS. 9A and 9B show an exit pupil 900 with an aspect ratio ofapproximately 3.5:1. The microlens array 910 uses hexagonal packing. Theaxes of the hexagonal lattice are rotated 30° relative to the long axisof the exit pupil 900. In FIG. 9A, A is the diameter of an exit pupil920 that results in an optimal packing of circular disk images onto thesensor without overlap, given a hexagonal microlens array and a focallength. When the long axis of the exit pupil is rotated 30° relative tothe hexagonal axes of the microlens array, the dimensions of arectangular exit pupil that results in optimally packed disk images hasa width of SQRT(3)*A and a height of A/2. FIG. 9B conceptually shows thepacking of the rectangular disk images 930 relative to the hexagonallypacked microlens array 910.

FIGS. 11A and 11B show further exemplary embodiments with hexagonallypacked microlens arrays and disk images, consistent with k=1 in thetable above. Specifically, FIGS. 11A and 11B show another way toimplement the system in FIGS. 9A and 9B. Instead of increasing thedimension of the exit pupil 1120 from A to SQRT(3)*A as in FIGS. 9A and9B, the width may be fixed to A, while the f/# of the microlens may beincreased by SQRT(3) while the diameter of each microlens 1130 in themicrolens array 1100 is decreased by the same factor. In this way, thedensity of the microlens array 1100 (within a fixed area) is increasedand thus the overall spatial resolution can be improved.

FIG. 12 shows a hexagonally packed microlens array 1200 and disk images,consistent with k=2 in the table above. FIG. 12 conceptually shows thepacking of the rectangular disk images 1210 relative to the hexagonallypacked microlens array 1200, when the MLA-exit pupil rotation isapproximately 19° and the aspect ratio of the exit pupil isapproximately 8:1.

In at least one embodiment where the exit pupil has a high aspect ratio,the image sensor-to-exit pupil rotation may be 0°. In these embodiments,the axes of the disk images may align with the axes of the photosensor.

In at least one embodiment where the exit pupil has a high aspect ratio,the f/# of the microlens array may be slightly smaller than the f/# thatmay result in ideal tessellation of disk images with no gaps or overlap.The slightly smaller f/# may introduce small dark regions between diskimages, and may reduce crosstalk between neighboring disk images whenthe light is captured by the photosensor.

One skilled in the art will recognize that the embodiments describedherein may be extended to include any type of microlens packing. Forexample, in addition to the square, rectangular, hexagonal, andelliptical packing arrangements described herein, the microlens packingmay be triangular, diamond shaped, or in any other pattern.

Discontinuous Aperture

In some embodiments, a disjointed or discontinuous exit pupil may beused. Embodiments of this type may be preferred, for example, when alarger optical baseline is desired, but decreasing the depth-of-field ofthe subviews is not.

FIG. 18A conceptually shows a discontinuous exit pupil having twosections 1810 separated by a distance 1820 equal to the sum of thewidths of each section. Each of the two sections 1810 may have a width1830 equal to d, and the center gap may have a width equal to thedistance 1820, which may be 2*d. In other embodiments, the two sections1810 may have a total width that sums to 2d, where one section may havea width equal to d+x and the other a width equal to d−x. Compared to asystem with Nu angular samples, a given depth-of-field in each subview,and an optical baseline of 2*d, this embodiment may have the same numberof angular samples, the same depth-of-field in each subview, and doublethe optical baseline.

FIG. 18B shows how segments 1860 of a disk image created from a squarepacked microlens array 1850 and an exit pupil 1800 of this type maytessellate when the microlens array 1850 uses square packing and theMLA-to-exit pupil rotation is 26.6°. Notably, a discontinuous exit pupilthat is divided into two parts like the exit pupil 1800 may be used inconjunction with various MLA-to-exit pupil rotations and packingpatterns to increase or reduce the aspect ratio.

FIGS. 14A and 14B show another embodiment. In this example, the exitpupil 1400 contains three discontinuous sections: a wide center section1410, and two narrow end sections 1420. FIG. 14B shows how the segments1440 of the disk image tessellate relative to a microlens array 1430using hexagonal packing. In other embodiments, the relative sizes ofeach section of a pupil with three discontinuous sections may beadjusted relative to each other to provide other tessellation patterns.

FIGS. 19A and 19B show two exemplary embodiments on a square packedmicrolens array, using an exit pupil that is discontinuous in twodimensions. Specifically, FIG. 19A illustrates segments 1900 of a diskimage on a microlens array 1910, in which each segment 1900 includesfour isosceles triangles oriented outward from an empty square. Theempty square of each segment 1900 may be filled with the isoscelestriangles of neighboring segments 1900. FIG. 19B illustrates segments1950 of a disk image on a microlens array 1960, in which each segment1950 includes a square and four trapezoids oriented outward from thesquare, which surrounded by a larger empty space with a square boundary.The empty space of each segment 1950 may be filled with the trapezoidsof neighboring segments 1900.

One skilled in the art will recognize that the above embodiments areonly a few of the many possible optical configurations that may use adiscontinuous exit pupil. Discontinuous pupils may have any number ofsections, which may be arranged in a wide variety of one-dimensionaland/or two-dimensional patterns.

Sensor Alignment

In some of the embodiments described above, accurate and specificMLA-to-exit pupil rotation may be required for the disk images toproperly tessellate on the photosensor without overlapping. In general,the image sensor-to-exit pupil rotation and the MLA-to-sensor rotationmay be unspecified.

In some embodiments, the image sensor-to-exit pupil rotation is 0°, orsubstantially 0°. Alignment of the exit pupil axes with the axes of thephotosensor may reduce crosstalk between disk images on the photosensor,and/or reduce the need for any “dead zone” allocated between diskimages. In some embodiments, dead zones may be used to reduce crosstalkbetween disk images, typically, by slightly reducing the f/# of themicrolens array relative to the objective lens.

Controllable Aperture

In at least one embodiment, a controllable and/or adjustable aperturemask may be used in conjunction with the embodiments described above anda lens with a low f/# compared to the f/# of the microlens array. FIGS.20A through 20D conceptually show an embodiment that may use a varietyof masks to create a light-field camera with different configurableproperties, but without changing the microlens array, photosensor,and/or any additional aspect of the objective lens. In this exemplaryembodiment, the microlens array may use square packing and have an f/#substantially equal to X.

FIG. 20A conceptually shows a fully “open” aperture 2000, which may havea lower f/# than the microlens array. When the mask is absent or fullyopen, the segments of the disk image in this exemplary embodiment mayoverlap, and the captured light-field data may not be usable.

FIG. 20B shows the aperture 2000 with a square mask 2020 applied.

With the square mask 2020 applied, the f/# of the objective lens may besubstantially equal to X, and the segments of the disk image maytessellate on the photosensor with little dead space and/or overlap. AMLA-to-exit pupil rotation of 0° may be used. When the square mask 2020is applied, the light-field camera may perform like a standard plenopticlight-field camera, with substantially equivalent sampling in bothangular dimensions.

FIG. 20C shows the aperture 2000 with a mask 2040 having a 2:1 aspectratio applied at or near the aperture plane. In order for the segmentsof the disk image to tessellate without overlap on the surface of thephotosensor, an MLA-to-exit pupil rotation of 45° may be applied.

FIG. 20D shows the aperture 2000 with a mask 2060 having a 5:1 aspectratio applied at or near the aperture plane. In order for the segmentsof the disk image to tessellate without overlap on the surface of thephotosensor, an MLA-to-exit pupil rotation of 26.6° may be applied.

One skilled in the art will recognize that the controllable aperture cansupport any of the embodiments described in this document, and manyothers. For example, any of embodiments described in the Exit Pupilswith High Aspect Ratio section are supported, for any value of k.

Notably, while the shape of the portion of the mask that allows light topass may vary substantially, the total area of such portions mayadvantageously be substantially identical. As the number of themicrolenses in the microlens array may be unchanging, and the size ofthe photosensor may be unchanging, the total area allocated to each diskimage such that there is minimal dead space and/or overlap, may also beunchanging.

Further, while the exemplary embodiment shows how aperture masks may beused to vary the aspect ratio of an exit pupil on a system with a squarepacked microlens array, it should be clear to one skilled in the artthat the concept naturally extends to alternative microlens array usingalternative packing (for example, hexagonal, rectangular, elliptical,etc.) and/or alternative exit pupil designs (for example, adiscontinuous aperture).

In at least one embodiment, an LCD panel may be placed at or near theaperture plane. The LCD panel may allow light to pass through certainareas of the aperture, while blocking light passage through other areas.As LCD panels may be manufactured with very high resolutions, nearlyarbitrary mask shapes may be generated. The LCD panel mask may becontrolled in any manner, including but not limited to manual control,automatic control and/or control via a user interface on the camerasystem.

In at least one embodiment, mechanical masks may be inserted into and/orremoved from the aperture plane. The mechanical masks may be made fromany suitable material, for example thin sheets of black plastic or blackanodized aluminum. The material may be cut to remove the portionsthrough which light is to pass. In one embodiment, the objective lensmay have a slit in the side that allows users to manually apply and/orremove mechanical aperture masks. In another embodiment, the objectivelens may include one or more mechanical masks that may be automaticallychanged. The mechanical mask insertion and/removal may be controlled inany manner, including manual control, automatic control and/orcontrolled by a user interface on the camera system.

In some embodiments, various masks, such as the square mask 2020, themask 2040, and the mask 2060, may be provided in a single camera throughthe use of a movable element on which all of the masks are located. Forexample, a rotatable disk may have masks that can be selectively rotatedinto alignment with the aperture. Alternatively, a rectangular strip mayhave masks arranged in a linear fashion such that translation of thestrip can selectively move the masks into alignment with the aperture.Alternatively, various movable elements can be combined to provide amask with a changeable shape. For example, four rectangular plates maybe movable toward or away from the center of the aperture to effectivelyprovide the square mask 2020, the mask 2040, and/or the mask 2060. Thus,the user of the light-field camera may select the appropriate mask foreach shot.

Alternatively, other approaches can be used for selectively allowinglight to pass through certain areas of the aperture, while blockinglight passage through other areas. Any known system for providingvariable light passage may be used.

Processing

Processing of the light-field images from the embodiments listed abovemay be performed using the same algorithms and techniques generally usedto process light-field images. In at least one embodiment, some changesare made to such algorithms and techniques so that they better suit thearchitectures described herein.

During processing of light-field images, each pixel on the photosensor(having raster position (s, t) in the light-field image) may be mappedto a light-field coordinate (x, y, u, v).

(x,y,u,v)=f(s,t)

In a system using one of the embodiments described herein, a maskingfunction may be used to help create this mapping. In one embodiment, themapping of raster coordinates to light-field coordinates may beperformed with an algorithm such as the following:

diskImageList = lightField.getDiskImageList( ) for each diskImage indiskImageList  x = diskImage.center.s / lightField.raster.width  y =diskImage.center.t / lightField.raster.height  bb =diskImage.boundingBox  for (t=bb.startT; t<bb.endT; t++)  for(s=bb.startS; s<bb.endS; s++) if (diskImage.mask.contains(s, t)  deltaS= diskImage.center.s − s  deltaT = diskImage.center.t − t  u =lightField.deltaPixelsToAngularCoord(deltaS)  v =lightField.deltaPixelsToAngularCoord(deltaT)

where:

-   -   lightField is a light-field image object. The object has a list        of disk images, determined previously (for example, during a        calibration process) or calculated as needed (for example, from        a geometric and/or optical model of the camera system).    -   x and y are the spatial light-field coordinates, in a normalized        0 to 1 range.    -   diskImage is a disk image object. Each disk image knows the        location of its center coordinate, contains a bounding box        outside of which are no pixels in the disk image, and a masking        test function that returns true if a raster coordinate contains        image data associated with the disk image.    -   u and v are the angular light-field coordinates, centered at 0.    -   deltaPixelsToAngularCoord is a conversion function that converts        from linear pixel offsets to an angular coordinate. The input is        the offset of a raster coordinate from the center location of        the disk image containing that coordinate along a single        dimension.

Once the light-field coordinates have been generated, processing of thelight-field image may be carried out in a manner similar to those setforth in prior art descriptions of light-field image processing. Theresulting light-field image may have enhanced angular resolution alongone dimension, which may facilitate and/or enhance the manner in whichthe light-field image may be used.

The above description and referenced drawings set forth particulardetails with respect to possible embodiments. Those of skill in the artwill appreciate that the techniques described herein may be practiced inother embodiments. First, the particular naming of the components,capitalization of terms, the attributes, data structures, or any otherprogramming or structural aspect is not mandatory or significant, andthe mechanisms that implement the techniques described herein may havedifferent names, formats, or protocols. Further, the system may beimplemented via a combination of hardware and software, as described, orentirely in hardware elements, or entirely in software elements. Also,the particular division of functionality between the various systemcomponents described herein is merely exemplary, and not mandatory;functions performed by a single system component may instead beperformed by multiple components, and functions performed by multiplecomponents may instead be performed by a single component.

Reference in the specification to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may include a system or a method for performing theabove-described techniques, either singly or in any combination. Otherembodiments may include a computer program product comprising anon-transitory computer-readable storage medium and computer programcode, encoded on the medium, for causing a processor in a computingdevice or other electronic device to perform the above-describedtechniques.

Some portions of the above are presented in terms of algorithms andsymbolic representations of operations on data bits within a memory of acomputing device. These algorithmic descriptions and representations arethe means used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. An algorithm is here, and generally, conceived to be aself-consistent sequence of steps (instructions) leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical, magnetic or optical signals capable of being stored,transferred, combined, compared and otherwise manipulated. It isconvenient at times, principally for reasons of common usage, to referto these signals as bits, values, elements, symbols, characters, terms,numbers, or the like. Furthermore, it is also convenient at times, torefer to certain arrangements of steps requiring physical manipulationsof physical quantities as modules or code devices, without loss ofgenerality.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“displaying” or “determining” or the like, refer to the action andprocesses of a computer system, or similar electronic computing moduleand/or device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain aspects include process steps and instructions described hereinin the form of an algorithm. It should be noted that the process stepsand instructions of described herein can be embodied in software,firmware and/or hardware, and when embodied in software, can bedownloaded to reside on and be operated from different platforms used bya variety of operating systems.

Some embodiments relate to an apparatus for performing the operationsdescribed herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computing deviceselectively activated or reconfigured by a computer program stored inthe computing device. Such a computer program may be stored in acomputer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs,magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, flash memory, solid state drives,magnetic or optical cards, application specific integrated circuits(ASICs), and/or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Further, thecomputing devices referred to herein may include a single processor ormay be architectures employing multiple processor designs for increasedcomputing capability.

The algorithms and displays presented herein are not inherently relatedto any particular computing device, virtualized system, or otherapparatus. Various general-purpose systems may also be used withprograms in accordance with the teachings herein, or it may proveconvenient to construct more specialized apparatus to perform therequired method steps. The required structure for a variety of thesesystems will be apparent from the description provided herein. Inaddition, the techniques set forth herein are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement thetechniques described herein, and any references above to specificlanguages are provided for illustrative purposes only.

Accordingly, in various embodiments, the techniques described herein canbe implemented as software, hardware, and/or other elements forcontrolling a computer system, computing device, or other electronicdevice, or any combination or plurality thereof. Such an electronicdevice can include, for example, a processor, an input device (such as akeyboard, mouse, touchpad, trackpad, joystick, trackball, microphone,and/or any combination thereof), an output device (such as a screen,speaker, and/or the like), memory, long-term storage (such as magneticstorage, optical storage, and/or the like), and/or network connectivity,according to techniques that are well known in the art. Such anelectronic device may be portable or nonportable. Examples of electronicdevices that may be used for implementing the techniques describedherein include: a mobile phone, personal digital assistant, smartphone,kiosk, server computer, enterprise computing device, desktop computer,laptop computer, tablet computer, consumer electronic device,television, set-top box, or the like. An electronic device forimplementing the techniques described herein may use any operatingsystem such as, for example: Linux; Microsoft Windows, available fromMicrosoft Corporation of Redmond, Wash.; Mac OS X, available from AppleInc. of Cupertino, Calif.; iOS, available from Apple Inc. of Cupertino,Calif.; Android, available from Google, Inc. of Mountain View, Calif.;and/or any other operating system that is adapted for use on the device.

In various embodiments, the techniques described herein can beimplemented in a distributed processing environment, networked computingenvironment, or web-based computing environment. Elements can beimplemented on client computing devices, servers, routers, and/or othernetwork or non-network components. In some embodiments, the techniquesdescribed herein are implemented using a client/server architecture,wherein some components are implemented on one or more client computingdevices and other components are implemented on one or more servers. Inone embodiment, in the course of implementing the techniques of thepresent disclosure, client(s) request content from server(s), andserver(s) return content in response to the requests. A browser may beinstalled at the client computing device for enabling such requests andresponses, and for providing a user interface by which the user caninitiate and control such interactions and view the presented content.

Any or all of the network components for implementing the describedtechnology may, in some embodiments, be communicatively coupled with oneanother using any suitable electronic network, whether wired or wirelessor any combination thereof, and using any suitable protocols forenabling such communication. One example of such a network is theInternet, although the techniques described herein can be implementedusing other networks as well.

While a limited number of embodiments has been described herein, thoseskilled in the art, having benefit of the above description, willappreciate that other embodiments may be devised which do not departfrom the scope of the claims. In addition, it should be noted that thelanguage used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter. Accordingly,the disclosure is intended to be illustrative, but not limiting.

1.-29. (canceled)
 30. A light-field camera comprising: an apertureconfigured to receive incoming light and having a rectangular exitpupil; an image sensor; a microlens array disposed between the apertureand the image sensor, wherein the microlens array comprises a pluralityof microlenses arranged in a plurality of rows, each row arranged at anon-zero acute angle relative to the rectangular exit pupil; and whereinthe image sensor is configured to generate light-field data based on theincoming light received through the microlens array.
 31. The light-fieldcamera of claim 30, wherein the plurality of microlenses comprises aplurality of rectangular microlenses.
 32. The light-field camera ofclaim 30, wherein the plurality of microlenses comprises a plurality ofcircular microlenses.
 33. The light-field camera of claim 30, whereinthe plurality of microlenses comprises a plurality of hexagonalmicrolenses.
 34. The light-field camera of claim 30, wherein a width ofthe rectangular exit pupil is greater than a length of the rectangularexit pupil.
 35. The light-field camera of claim 30, wherein therectangular exit pupil is shared and oriented, relative to the microlensarray, such that the incoming light received through the microlens arrayforms a tessellated pattern at the image sensor.
 36. The light-fieldcamera of claim 30, further comprising: a main lens through which theincoming light is to pass through prior to redirection by the microlensarray.
 37. The light-field camera of claim 30, further comprising: amasking system configured to position at least one of a plurality ofmasks proximate the aperture, wherein the masks define a plurality ofexit pupils having a plurality of different shapes, each of whichdefines an exit pupil for the aperture, the plurality of exit pupilsincluding the rectangular exit pupil.
 38. A light-field cameracomprising: an aperture configured to receive incoming light and havinga rectangular exit pupil; an image sensor; a microlens array disposedbetween the aperture and the image sensor, wherein the microlens arrayis rotated by a non-zero acute angle relative to the rectangular exitpupil; and wherein the image sensor is configured to generatelight-field data based on the incoming light received through themicrolens array.
 39. The light-field camera of claim 38, whereinmicrolens array comprises one of: a plurality of rectangularmicrolenses; a plurality of circular microlenses; and a plurality ofhexagonal microlenses.
 40. The light-field camera of claim 38, wherein awidth of the rectangular exit pupil is greater than a length of therectangular exit pupil.
 41. The light-field camera of claim 38, whereinthe rectangular exit pupil is shared and oriented, relative to themicrolens array, such that the incoming light received through themicrolens array forms a tessellated pattern at the image sensor.
 42. Thelight-field camera of claim 38, further comprising: a main lens throughwhich the incoming light is to pass through prior to redirection by themicrolens array.
 43. The light-field camera of claim 38, furthercomprising: a masking system configured to position at least one of aplurality of masks proximate the aperture, wherein the masks define aplurality of exit pupils having a plurality of different shapes, each ofwhich defines an exit pupil for the aperture, the plurality of exitpupils including the rectangular exit pupil.
 44. A computer-implementedmethod comprising: generating, at a processing system, a two-dimensional(2D) image from captured light-field data, the light-field datacomprising a first spatial dimension, a second spatial dimension, afirst angular dimension, and a second angular dimension, and wherein afirst resolution of the light-field data in the first angular dimensionis larger than a second resolution of the light-field data in the secondangular dimension.
 45. The method of claim 44, further comprising:capturing the light-field data using a light-field camera having anaperture configured to receive incoming light and having a rectangularexit pupil, an image sensor, and a microlens array disposed between theaperture and the image sensor, wherein the microlens array comprises aplurality of microlenses arranged in a plurality of rows, each rowarranged at a non-zero acute angle relative to the rectangular exitpupil.
 46. A non-transitory computer readable medium embodying a set ofexecutable instructions to: generate, at a processing system, atwo-dimensional (2D) image from captured light-field data, thelight-field data comprising a first spatial dimension, a second spatialdimension, a first angular dimension, and a second angular dimension,and wherein a first resolution of the light-field data in the firstangular dimension is larger than a second resolution of the light-fielddata in the second angular dimension.
 47. The non-transitory computerreadable medium of claim 46, wherein the set of executable instructionsfurther are to: capture the light-field data using a light-field camerahaving an aperture configured to receive incoming light and having arectangular exit pupil, an image sensor, and a microlens array disposedbetween the aperture and the image sensor, wherein the microlens arraycomprises a plurality of microlenses arranged in a plurality of rows,each row arranged at a non-zero acute angle relative to the rectangularexit pupil.